eks-prow-build-cluster: Use dedicated Managed Node Groups (MNGs) per Availability Zone (AZ) #6320
Conversation
Signed-off-by: Marko Mudrinić <mudrinic.mare@gmail.com>
Signed-off-by: Marko Mudrinić <mudrinic.mare@gmail.com>
Signed-off-by: Marko Mudrinić <mudrinic.mare@gmail.com>
Signed-off-by: Marko Mudrinić <mudrinic.mare@gmail.com>
k8s-ci-robot
added
cncf-cla: yes
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: upodroid, xmudrii The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
cancel the hold when you are ready to merge |
k8s-ci-robot
added
the
do-not-merge/hold
|
Let's get this merged and I'll create a new PR for follow-ups |
k8s-ci-robot
removed
the
do-not-merge/hold
|
|
||
| cluster_version = var.node_group_version_us_east_2a | ||
|
|
||
| taints = var.node_taints_build |
There was a problem hiding this comment.
FYI - you can use eks_managed_node_group_defaults to set the default values you wish to use across all of the managed nodegroups, and then any specific settings unique to that nodegroup can be set in the nodegroup definition. You can also set the defaults and still override the value in the nodegroup definition as well
So roughly speaking, something like the following might help cut down on the copy+pasta across nodegroup definitions:
eks_managed_node_group_defaults = {
use_name_prefix = true
taints = var.node_taints_build
labels = var.node_labels_build
... anything else that you want common across the nodegroups
}
There was a problem hiding this comment.
That's neat, thanks for pointing it out! I'll keep it in mind and see if we can make use of it the next time we do some rollout
xref #6303 (see for more details and context)
As per recommendations received from the AWS support and @tzneal, we're replacing blue and green node groups with node groups per AZ. In other words, old node groups had instances in all three AZs. Now, we have a dedicated node group for each AZ. This is a short-term solution to fix the stability issues that we're facing.
This has been successfully rolled out to canary, I'll do prod rollout once this PR is reviewed and merged.
Notes:
countbecause that would make rolling out upgrades too complicatedFollow-ups:
/assign @upodroid @ameukam @dims